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ABSTRACT 

In response to high demand for 
Engl ish~as-a~Second~Language (ESL) instruction among the community's 
Chinese-speaking population, the Quincy School Community Council 
(Massachusetts) developed the Take And Give (TAG) program, in which 
graduates of the highest-level ESL classes were trained as peer 
tutors for students at the lowest instructional level. The program's 
goal is to help students at the introductory level, at which there is 
greatest demand for instruction, bypass level 1 classes and proceed 
to level 2 classes. TAG students study at home using a videotape 
series and meet with tutors bi-weekly to review, supplement, and 
practice what they have learned. The first 8-week tutor training 
cycle and student recruitment began in January 1992. The report 
details the evaluation of both the tutor training program and the 
student instructional program, and includes a program participant 
profile. Evaluation of student ESL learning focuses on student 
background, motivation, learning strategies, and classroom 
performance. Further development of an ongoing program evaluation 
system is also discussed. Appended materials include data on TAG 
participants and comparison groups, classroom observation data, and 
TAG program participant retention data. (MSE) (Adjunct ERIC 
Clearinghouse on Literacy Education) 
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Executive Summary 



TAG Background 

By the summer of 1990, the Quincy School Community Council's Adult 
English as a Second Language program was facing waiting lists of 
over 1000 new immigrants (mostly Chinese) for its English classes. 
In response, they created the Take and Give (TAG) Program, in which 
peer tutors, graduates of the top level English classes, were 
trained to work with learners on the waiting lists. The TAG 
learner participants study in their own homes a videotape series 
called "Practical English," developed specifically for Chinese 
learners of English. The tutors meet with pairs of TAG learners 
every two weeks to review, supplement, and practice English lessons 
associated with the tapes. The goal is to prepare participants for 
entry into a Level 2 English class within six months to a year, 
thus bypassing the Level 1 classes where the greatest logjam 
exists. The TAG Program received its first funding and began 
operation in late 1991 with planning and, in January 1992, the 
first 8-week training cycle for tutors and recruitment of TAG 
learners . 

The Evaluation 

By the summer of 1992, initial indicators were that the program was 
successful, and that it should continue, develop, perhaps be 
replicated by others, and that a formal evaluation should^ be 
conducted to support those processes. As a result, funding was 
obtained by a grant from the U.S. National Institute for Literacy 
to continue support and development of the program as well as to 
conduct an evaluation, for which Development Assistance Corporation 
of Dover NH was selected as the subcontractor. 

The evaluation design was intended to be useful as an ongoing 
program analysis and feedback system to QSCC's staff as well as a 
conventional external evaluation. Two features resulted from this 
intention: one, DAC involved TAG staff in instrument development 
and trained them to conduct data collection activities throughout 
the project to build QSCC's capacity in evaluation; and two, DAC 
and TAG staff jointly worked through an adaptation and application 
of the Discrepancy Evaluation Model for TAG, yielding a framework 
for an integrated system of evaluation and program analysis and 
development . 

Out of this DEM framework, we focused the evaluation on both 
processes and outcomes of the tutor training component and the 
tutoring components. To do so, we conducted the following data 
collection activities : 

. interviews with 17 of the 37 tutors, after they had 
tutored for at least two months 

native language interviews with 36 of the 109 TAG 
participants, after they had been tutored for at least 
two months 
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• interviews with the three teachers of AESL Level 2 
classes into which TAG "graduates" entered 

• administration of the BASIC ENGLISH SKILLS TEST (BEST) to 
TAG and nonTAG students in Level 2 classes, pre and post 

• observations of the tutor training program, two different 
cycl es 

• observations of the AESL Level 2 classes into which TAG 
"graduates " entered 

• collection and analysis of record/ file data on 
participant characteristics and program retention 

Participant Profile 

The general trend among TAG participants, compared to AESL Level 1 
and 2 waiting listjnembers, and Level 2 enrollees, is that they are 
slightly older, less educated, and more often married and with 
children than their counterparts in the other groups. These are 
the people who might find it more difficult to schedule a formal 
class than their younger, more educated, single or fewer children 
fellows would. Thus, TAG affords an important opportunity for them 
to begin English learning in a manner more suited to their life 
circumstances . TAG is not used just by the 'immigrants with the 
characterization above, however, for the entire spectrum of 
demographic characteristics are represented by the participants . 

Evaluation of the Tutor Training Process and Outcomes 

In the tutors' views, the tutor training program was effective in 
making them feel prepared to tutor. They highlighted the 
achievement of the two main program objectives : better English 
pronunciation and teaching skills. While the best features of the 
training program were seen as learning good teaching skills, having 
a good model in their trainer, ways of motivating students, and 
speaking with more experienced tutors, they also suggested that 
improvements might be made in having more time to learn, building 
more confidence, learning more American usages of words, doing more 
reading, and learning techniques particularly useful for adult 
students . There was also a repeated theme of desiring that more 
tutoring practice be part of the training program. The materials 
used in the tutor training were unanimously seen as very helpful, 
although a few found the reporting materials too hard to use. 

The tutees generally rate their tutors very well, with only a few 
commenting that they don't understand all they would like to from 
the sessions . After some experience tutoring, some tutors 
indicated the need for help in making more and better practice 
sheets for their students . Some tutors apparently do this to a 
great degree and others do not, a feature mentioned by both tutors 
and tutees. It was also suggested that periodic say, bimonthly 
— sessions be held for the tutors to come together to discuss 
their ongoing experiences tutoring. 
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The tutees almost unanimously rated the videotapes and supportive 
materials as ''just right" in level of difficulty for the TAG 
participants to learn English, with a few indicating they were 
either "too easy" or "too hard." The tutors, however, were almost 
equally split as to whether the tapes were "too easy, " "just 
right, " or "too hard. " Some areas for improvement noted included 
having longer dialogues and lessons, more subtitles for the 
teachers' speech on the tapes, books to accompany the tapes, more 
conversations about daily life, slower speed, and updating the 
skits. 

Evaluation of Participants' English Learning 

The immigrants enrolled in the TAG Program generally had some 
minimal training in English before their U.S. arrival, and a few 
months of English study in the U.S. before participating in the TAG 
Program. They report being able to use very little English upon 
arrival, and they made little progress in English usage before TAG. 
After two or more months of TAG tutoring, however, they report a 
substantial increase in their English vocabulary, speaking with 
more detailed phrases, and usdng longer and more complete 
sentences . In addition, their confidence in using English is 
reportedly increased substantially by this point, even after little 
increase in earlier learning stages. 

The main motivations for learning English are economic — to get an 
initial or better job and to conduct their shopping. Other uses of 
English are present but secondary. Moreover, the TAG students in 
AESL classes are reported by their teachers to work somewhat harder 
than other students, possibly indicating higher levels of 
motivation. 

The majority of TAG participants found that the English levels used 
in the main tutoring tool, the videotapes and supporting materials, 
were "just right" to help them learn well, with a few finding them 
either "too difficult" or "too easy." In addition to these formal 
course materials, however, other television programs and movies 
were widely reported to be of additional assistance in learning 
English, along with talking to their children who were learning 
English in school, features which the TAG Program might attempt to 
capitalize on in future developments. 

Once the TAG students enter AESL Level 2 classrooms, they appear 
able to perform well. Interviews with the teachers indicated that, 
while their other students generally respond in class at a mid- 
level of confidence and ability, the TAG students varied widely, 
from speaking without hesitation at any opportunity to almost never 
attempting to speak. After a few months in the classes, however, 
the TAG students appear to "catch up" in response level to the 
norm. 

As judged by standardized English test scores, the TAG students 
gained about as much in English literacy as the other students. 
The non-TAG students, however, showed gains in oral English skills 
at about twice the level of the TAG students. This latter result 
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would worth examining further. Using the outside observers' 
ratings of English performance in the classrooms as the criterion, 
however, the TAG students performed only very slightly lower in 
English level than the non-TAG students, and both groups gained at 
about the same rates. 

Conclusions 

A. The TAG Program is clearly an efficient opportunity for 
immigrants to start formally studying English sooner than available 
classroom opportunities allow. This is especially true for those 
who fit into demographic groups with the greatest scheduling and 
other logistical barriers. 

B. The tutor training appears to be successful in developing 
English and tutoring skills to at least an adequate level for 
effective tutoring. 

C. The videotapes and supportive written materials appear to be at 
the appropriate level of challenge for most of the tutees, and the 
tutors appear to use them well in teaching English. 

D. TAG appears to have provided a substantial boost to the 
participants' English learning and confidence in English usage. 

E. TAG participants who enter AESL Level 2 classes progress as well 
as their non-TAG classmates in English literacy skills, but not 
nearly as well in oral language skills. 

F. The tutor competency, monitoring, and ongoing improvement 
component of the program could be better formalized. 

G. TAG staff evaluation capability has been developed, but its more 
complete establishment appears blocked by time availability or 
commitment to evaluation activities. 

For an amplified discussion of each conclusion above, see Section 
VII of the full report. Overviews of the program components 
outlined in the DEM framework and of the findings for each are 
presented in Figures 2 and 3 of the full report. 
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I. INTRODUCTION: Background of the TAG Program 



In the summer of 1990, the Adult English as a Second Language 
(AESL) program staff of Quincy School Community Council, Inc. 
(QSCC) began creating their answer to the long waiting lists of new 
Asian (mostly Chinese) immigrants in need of English instruction. 
In addition to the 500 adult students studying English as a Second 
Language (ESL) in regular classrooms at QSCC, over 1000 were on 
westing lists to enter classes* 1 By the spring of 1991, em 
innovative design, called the TAKE AND GIVE (TAG) Program, was 
ready and funding sought. Initial funding of $220,000 for three 
years was provided by two private foundations: The Harcourt General 
Charitable Foundation/The Neiman Marcus Group and The Jessie B. Cox 
Charitable Trust. 

Ongoing 8-week training cycles of about six AESL program 
graduates per cycle began in January 1992. When trained, these 
peer tutors work with newcomers using a videotaped series of ESL 
lessons called "Practical English." Since the greatest logjam of 
waiting list students was for Level 1 (basic survival English) , the 
TAG Program had a goal to get learners into Level 2 and, therefore, 
off the logjam part of the waiting lists by eliminating their need 
for Level 1 classes. 

The tutors, typically graduates of QSCC ' s Level 6 (the top 
level) classes, through the training sessions, continue to learn 
English as well as tutoring methods. At the completion of 



These facts, as well as other background information on TAG'S evolution, are 
presented in "Breaking the Waiting List Logjam: Training Peer Tutors for ESL, " by 
Roger Hooper, in All Write News of the Adult Literacy Resource Institute, Boston, 
MA, Volume VIII, No. 6, May/June 1992. 
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training, the tutors are matched by TAG staff with two or more 
students to meet twice a month to review the video lessons and 
supplementary worksheets, and work on other activities. The TAG 
students work at their own pace on the video lessons and worksheets 
between tutoring sessions , depending on their schedules and 
abilities. Having tutors with similar backgrounds to their own and 
success at English learning is considered beneficial to students, 
and peer tutors can continue learning and find gratification from 
helping others. 

By the summer of 1992, it appeared that the TAG Program was 
initially successful and could continue, develop, and perhaps be 
replicated by other ESL providers. A search of the U.S. Department 
of Education's ERIC (Educational Resources Information Center) 
system at this time indicated no other similar programs. The TAG 
Program indeed appeared to be a ground-breaking effort. Further 
funding was then sought to continue implementation of the TAG 
Program and to conduct a comprehensive evaluation in order to 
improve it, to report on its successes, and to disseminate it as a 
model. Having gained funding from the National Institute for 
Literacy, as well as continued support from the private 
foundations, QSCC was ready to subcontract an evaluator. 
Development Assistance Corporation (DAC) was chosen to evaluate the 
program and began working directly with TAG staff that fall. 

This report presents the original evaluation purposes and 
planned activities, how the evaluation was designed, what actual 
activities took place, what the findings and conclusions are, and 
what the next steps might be. 

a 
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II. EVALUATION DESIGN AND ACTIVITIES 



Purposes and Overview 

After considering possible evaluation models with TAG staff, 
DAC and TAG agreed to use an adaptation of the Discrepancy 
Evaluation Model (DEM) as a particularly beneficial framework for 
this type of program, as well as its "user-friendliness" to TAG 
staff in integrating the evaluation with the program development 
process. This framework and associated implementation processes 
were particularly important because of its requirement that the TAG 
staff be integrally involved in articulating the framework. In 
this case, a considerable amount of data collection had to be 
conducted by the bilingual TAG staff. Thus, their involvement in 
the developmental aspects of the evaluation helped their 
understanding of the data collection, .analysis, and evaluation use 
aspects. 

The DEM as applied to the TAG Program essentially consists of 
four steps, summarized as: 

1. articulate the TAG Program's goals and objectives; then 
outline the resources, implementation processes, and outcomes 
for each objective; 

2. document each element of the resources and implementation 
activities, and measure and analyze the outcomes (which may 
include context and baseline information such as pretest 
status) , for each objective; 

3 . analyze the discrepancies or the congruencies along the 
"track" from objectives, through resources, implementation, 
and outcomes , to identify those program elements that are 
working well or are not; and 

4. present the resulting evaluation to the program staff so 
that they can take steps to develop improvement plans for the 
weak areas and reinforce or expand the strong areas. 
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The DEM was built around the actual goals and objectives of 

the TAG Program, and several drafts were analyzed by DAC and TAG 

staff until a satisfactory model was completed (detailed in the 

next section) . Three main goals emerged from this analysis which 

will be examined closely in this report: 

GOAL #1: TUTOR TRAINING — TRAIN AESL PROGRAM GRADUATES AS 
ENGLISH TUTORS TO SUCCESSFULLY MATCH WITH APPROPRIATE LEARNERS 
FROM THE WAITING LIST 

GOAL #2: ENGLISH LEARNING TEACH ENGLISH TO CHINESE-SPEAKING 
IMMIGRANTS ON AESL WAITING LIST THROUGH THE TAG TUTORING 
PROGRAM TO FACILITATE THEIR ENTRY INTO THE AESL CLASSES 

GOAL #3: EVALUATE AND IMPROVE TAG 

We planned to collect and analyze data on the first two goals 
which would then be incorporated into the third goal. As part of 
this third goal, DAC would design an evaluation system in 
consultation with the TAG staff and conduct its activities for the 
first year, presenting a report with program analysis, conclusions, 
and recommendations at the end. Through this "conventional" 
evaluation, the TAG staff could consider those results for possible 
program development and improvement as part of their ongoing 
operation. Additionally, DAC would train TAG staff to carry out 
those methods of the evaluation which worked well and which fit 
appropriately into their program operation roles, so that a useful 
and reasonable adaptation of the evaluation could be continued 
internally after this formal contract ended. 

Evaluation Design 

Following several planning sessions, attended by TAG and DAC 
staff members, and the choice and adaptation of the DEM, specific 
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activities were selected and outlined to measure the outcomes and 
implementation processes of the two TAG Program goals. The 
processes include the interactive aspects of 1) the program 
elements, 2) the peer tutors, and 3) the students. The outcomes 
consist of a) individual outcomes such as the improvement of a 
student's English skills, and b) the general program outcomes such 
as how well each goal or objective is attained (see Figure 1) , 



Figure 1. Factors to Be Measured 



- PROGRAM CONSIDERATIONS 


o goals/objectives 

o tut or- training 

o intake/assessment 

o curriculum materials ' 

0 videotapes 

o tutor supervision 

o problem-solving 

o other resources 


PROGRAM 


OUTCOMES 


TAG STUDENTS 


TUTORS 


• English 

• attitudes 

• learning skills 

• satisfaction 

• individual outcomes 


• qualifications 

• certification 

• procedures 

• reporting 

• individual outcomes 



To ascertain whether the TAG goals and objectives are 
appropriate and have been achieved, we planned activities to 
measure characteristics of the program, tutors, and students, 
assess the students and tutor training, and examine other 
indicators of the program such as materials, videotapes, and other 
resources. Besides the BASIC ENGLISH SKILLS TEST (BEST), developed 
by the Center for Applied Linguistics, the instruments we chose 
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were developed by us specifically for use in the TAG Program, 
piloted at the Quincy School, and revised with TAG staff to best 
measure outcomes and processes . Six methods emerged as most 
feasible, taking into account the evaluation needs and resources: 

• TAG Tutor Interviews 

• TAG Participant Interviews 

• AESL Level 2 Teacher Interviews 

• BEST Test Administration 

• TAG Tutor Training Observations 

• AESL Classroom Observations 

• extraction of student data from records/ files . 

These activities, conducted by both DAC and TAG staff, are 
described in more detail in the next section. Results and analysis 
of the outcomes can be found in sections IV and V. The copies of 
instruments themselves are included another volume, the Training 
Manual . 

Evaluation Activities 

One of the major project activities, the backbone of the 
evaluation, involved developing the evaluation system. The 
intention is that this system will be transferred to the TAG 
operations as part of its normal functioning. The set of 
activities for developing the evaluation system included: a) 
development of the initial evaluation framework through applying 
the DEM to the TAG; b) developing a set of preliminary instruments 
and a data collection plan; c) pilot-testing and revising the 
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instruments; d) training bilingual TAG staff in the data collection 
instruments and methods; and e) collecting data and analyzing it. 

To develop the initial evaluation framework using the DEM, we 
first drafted an analytical description of the TAG Program based on 
our outsiders' knowledge of the program from written materials 
provided by the TAG staff, reviews of curriculum materials and 
/Sample videotapes, and from conversations with the staff . This 
analytical description outlined the key activities we saw occurring 
(or planned) within each program goal, and then the resources 
necessary to conduct each activity successfully and the expected 
outcomes for each activity. The draft was presented to the TAG 
staff for comment, then again revised by DAC. After two such 
internal staff critiques and external evaluator revisions, ^the 
final working version of the DEM was developed (see Figure 2) . 

DAC staff then reviewed the outcomes of the working DEM to 
assign potential types of measurement for each outcome, selecting 
from among: a) interviews with TAG participants, tutors, and AESL 
teachers; b) observations of the intake process, tutor training 
sessions, and AESL classes; c) English learning assessments; and d) 
records' retrieval. Proposed instruments with the outcomes to be 
measured by each were presented to and discussed with TAG staff 
before f inalization . Refinements from these discussions included 
the addition of information available in participants' records, 
clarifications that certain types of information might not be 
accessible through participant interviews, and translation of the 
participant interview form into Chinese by TAG staff, since it was 
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determined that this instrument would have to be administered in 
Chinese because of the low English levels of participants. In the 
cases of observation instruments, DAC was allowed to conduct 
exploratory observations of tutor training sessions and AESL 
classes in order to become acquainted in detail with those settings 
and operations. For the English assessment, we were assisted by a 
consultant with extensive experience in English testing for LEP 
populations, who reviewed several available tests and discussed 
them with the TAG staff. The consensus was to use the BEST. 

Pilot testing and revision^ of instruments were conducted by 
DAC staff. TAG staff were then trained by DAC in the 
administration of the BEST, the participant interview protocol, and 
AESL class observation. 

The following data collection activities were conducted: 

• TAG Tutor Interviews — conducted 17 interviews in 
English with tutors (of the 37 trained) ; these included 
6 of 24 from Cycles 1-4, 5 of 6 from Cycle 5, and 6 of 7 
from Cycle 6 . 

• TAG Participant Interviews conducted 36 interviews 
in Chinese with a sample of the 109 participants in 
Cycles 1-6. 

• AESL Level 2 Teacher Interviews conducted interviews 
with the three teachers who had TAG students in their 
classes . 

• BEST Administration — administered pretest in January 
and June to 39 students in two Level 2 classes, 16 TAG 
participants, 23 not TAG participants; administered 
pretest in August to 29 students in the two classes, 17 
TAG participants, 12 not TAG participants. 

• TAG Tutor Training Observations — conducted pilot 
tests of observation forms during the Cycle 6 training; 
conducted pre- and post-observations during the Cycle 7 
training . 
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• AESL Level 2 Classroom Observations — conducted pilot 
tests of the observation forms during Fall 1992; pre-, 
mid-, and post-observations during the January-June 1993 
term; and pre-observations during the July-December 1993 
term. 

• Record/ file Data Extraction -- pulled demographic 
characteristics from student database, cross tabulated; 
examined files on each tutee for retention information. 
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III. TAG PARTICIPANT PROFILE 



One of the important considerations in evaluating the effect 
of a program is what the participants are like. Additionally, when 
the participants' results are compared to those of non- 
participants, we also want to know how the characteristics of the 
two groups compare to each other. In the case of TAG, we can then 
try to determine if any differences in such results as English 
achievement between TAG participants and other AESL class students 
might be attributable to basic student differences and not 
necessarily the effect of the TAG" tutoring . 

To examine these issues , information on a number of 
demographic characteristics was retrieved from QSCC's data files on 
four groups of people as of March 1993: TAG participants, AESL 
Level 1 waiting list members, Level 2 waiting list members, and 
Level 2 enrollees. The data gathered included: years on the 
waiting list, gender, age, marital status, number of children, 
years of formal education, months of previous English study, time 
in the U.S., and income. All information of a changing nature 
(e.g., time in the U.S.) was specific to the time of registration 
with QSCC. Detailed information on each characteristic for the TAG 
enrollees and three comparison groups is presented in Appendix A. 

The TAG participants are generally characterized as follows. 
More than half (57 percent) have been on the AESL waiting list for 
less than one year before getting into TAG, alchough a third were 
on for almost three years (which is probably a one-time occurrence, 
since TAG only began within the past year, and there was a backlog 
of long-time waiting list members). By contrast, half or more of 
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the Levels 1 and 2 waiting list members and the Level 2 enrollees 
were on waiting lists for almost three years. Thus, TAG is clearly 
an opportunity to begin formal English study sooner than by waiting 
for an AESL class. 

While a slight majority (54 percent) of the TAG participants 
are in the 25-44 age category, this is substantially lower than the 
great majority (68 to 80 percent) of this age group in the other 
three comparison groups. The difference is made up for by the 35 
percent of TAG participants in the 45-59 age group, compared to 
lower numbers in the comparison groups (10, 16, and 22 percent). 
That is, TAG participants tend to be somewhat older than those who 
remain on class waiting lists or enroll in Level 2 classes. 

Perhaps consistent with the age differences, the formal 
education levels completed by TAG participants is slightly lower 
than those waiting for classes or enrolled in Level 2. Over two- 
thirds of TAG participants reported formal education of 4-6 years 
or 7-9 years. The Level 1 waiting list members have slightly more 
in the 7-9 years and 10 plus years categories, and the Level 2 
waiting list members and class enrollees have 80 and 58 percent 
respectively in those two categories. 

Since their time on the waiting list was comparatively short, 
the TAG students' time in the U.S. was also less than that of the 
other groups' members -- 77 percent of the TAG participants were in 
the U.S. for less than three years, compared to 40, 50, and 58 
percent for the other groups. 

Again consistent with the fact that the TAG students are 
somewhat older than the members of the other groups, a greater 
percentage of them are married compared to the other groups (85 
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percent, compared to 73 to 77 percent) . They also generally have 
more children. 

Finally, 71 percent of the TAG participants are female and 29 
percent male. The relative percentages of females and males are 
similar for each of the comparison groups. 

In practice, therefore, the general trend is that TAG seems 
most suited for immigrants who have come to the U.S. quite recently 
and v/ish to get started formally learning English sooner than 
classroom opportunities allow. They are also slightly older, less 
educated, and more often married and have children than other 
immigrants. This group may find i-t easier to begin their study of 
English in a less formal way than a traditional class and may also 
find the scheduling easier than the four days (or nights) a week 
for six months that the classes at QSCC meet. Thus, TAG appears s to 
provide an opportunity to overcome logistical scheduling barriers 
as well as those of possible intimidation by formal classes for 
less-educated immigrants. While that characterization exists, it 
is not universal, however, and clearly others who are younger, more 
educated, have been in the U.S. and on waiting lists longer have 
also taken advantage of TAG'S opportunity. 
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IV. EVALUATION OF THE TUTOR TRAINING PROCESS AND OUTCOMES 



The first goal of the TAG Program was to train AESL program 
graduates as English tutors, who could then conduct tutoring with 
Chinese immigrants on the class waiting list. To evaluate the 
extent to which that goal has been attained we interviewed 2 a 
sample of 36 participants selected from those 109 tutored in cycles 

1 through 6, and 17 of the 37 tutors trained in cycles 1 through 
6. 3 Interviews with three AESL teachers, whose then-current Level 

2 classes included some students who had completed the TAG Program, 
also provided some comments on -the overall tutoring program. 
Additionally, we observed the 6th cycle tutor training session 
about midway through its two-month term and the 7th cycle training 
program near its beginning and its end, and examined materials and 
curriculum time usage for the tutor training program. 

Observations of the Tutor Training Program 

The tutor training curriculum has three main objectives: 
learning in English skills, tutoring/ teaching skills, and using the 
TAG reporting system. A time analysis of the scheduled activities 
classified into those categories indicated that approximately 48% 
of time is spent on the tutoring/teaching skills, 44% on English 
learning, 2% on reporting, and 7% on other activities. These other 
activities include, for example, introductions among the group 



2 The interview questions were developed by DAC, refined with the assistance 
of the TAG staff, field^tested and finalized by DAC, and translated into Chinese and 
administered by TAG staff. The responses were then translated into English by TAG 
staff and presented to DAC for analysis. 

3 These 11 included a volunteer sample of six of the 24 trained in cycles 1 
through 4, which had been completed before we began the evaluation, and 11 of the 
13 trained during the course of this evaluation. 
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However, the translating did not necessarily come from the teacher, 
but another trainee or after some discussion between several people 
and some dictionary use. The only apparent disadvantage to the 
high use of English was that those who are less comfortable 
speaking English to their Chinese peers participated less in the 
verbal activities. 

Materials. The blackboard was rarely used as much of the work 
was done either verbally or on newsprint posted around the room. 
In addition to the agenda, many handouts with stories, phonetics, 
case studies, questions, pictures, etc. were used. The trainees 
accumulated these to use in their own teaching later when- they 
would have students. Actual record-keeping sheets were used for 
practice and to talk about the goals of tutoring. Sometimes the TV 
and VCR and videocassettes were used to view a lesson. Although 
not required by the teacher, several of the trainees would bring 
Chinese/English hand-held computer dictionaries for translating. 

Impressions. Although participation varied depending on topic 
and trainee comfort in speaking up, the tutor training sessions 
were lively and much material was covered. it was apparent that 
the trainers were establishing a trusting relationship with the 
potential tutors which they hoped would facilitate the process 
later if any problems came up. The case studies were based on real 
situations which the tutors might come across, and, as a result of 
the rapport with the trainer and practice in dealing with them, 
would know how to handle or who to turn to if stumped. Many of the 
trainees entered with a mixed feeling of excitement and fear that 
they had the opportunity to teach -- something most of them had 
never done, and none had done in SrQlish. The attitude of the 
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teacher and all the experience of working one-on-one with their 
peers would seem to be positive for the tutors' emulation. For 
example, the teacher never replied, "No, that's wrong," but instead 
made a suggestion or elicited better ideas for the subject at hand. 

Tutor Training and Skills: Tutor and Participant Perspectives 

The trainees' backgrounds in English, the foundation on which 
the TAG tutor training program builds, includes having initially 
learned English to a considerable degree in China or Hong Kong (or, 
in rare cases, another native country). For about a third of the 
tutors it was only in primary school, but for the other two-thirds 
it was in high school or college. Most tutor trainees also studied 
English at other community-based organization programs in the 
Boston area or at a local community college, in addition to haying 
studied in, and graduating from the highest level of QSCC's AESL 
program. 

These trainees indicate having moderate strengths in English 
writing, reading, speaking, listening, and grammar, all areas about 
equally balanced. Of all the types of English usage, almost all 
indicate a desire to increase their speaking ability, although 
about a third were also interested in improving their listening 
skills. In addition, most indicated a desire to continue their own 
schooling in the future, primarily to learn more English, although 
in about half wanted to study accounting, clerical and secretarial 
skills, and computer programming. Four individuals each wanted to 
study more specialized areas: teaching methods, nursing, medicine, 
and metal coating (an already practicing chemical engineer) . 

3 o 
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When asked if they were ready to tutor after the training, all 
respondents indicated that they were. Eleven of the 17 indicated 
no prior experience tutoring or teaching English. Of the six who 
had prior experience, this included teaching nursing, experience as 
a translator for Chinese-speaking clients in banks and hospitals, 
teaching assistant in a refugee camp, and teaching Chinese to high 
school students. 

The features these tutors felt were most salient about their 
preparation in the tutor training program included teaching skills 
and better English pronunciation, grammar, and writing, following 
the two main program objectives. Other less salient features 
mentioned were confidence and patience to help students through 
learning difficulties . 

One of the AESL teachers of TAG "graduates" suggested that an 
area for improvement would be to have the tutors go through a more 
intensive English learning and teaching methodology training, that 
these were needed to develop their confidence more and to gain more 
respect from the TAG students. This suggestion was made in the 
context of recognizing that the tutors do a "very good job" with 
the training and backgrounds they do have, but that this 
improvement would help the English learning of the TAG participants 
even more. The comment was also made in recognition of additional 
funding needed to provide that more intensive training, as the 
currently limited funds are being utilized to their fullest extent 
possible . 

The "best" parts of the training program mentioned by tutors 
were the teaching skills learned, having a trainer who modeled good 
teaching, ways to get in touch with students, the grammar, writing 

31 
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and pronunciation improvement gained, the good organization of the 
sessions, and the session (s) in which experienced tutors came in to 
talk with the trainees about tutoring, items mentioned that might 
have been better about the training included (followed by the 
frequency they were mentioned) : 

- more time to learn, training cycle too short (4) 
more English speaking practice (and less Chinese) (4) 

- more reading, writing and grammar (3) 

- building more confidence in tutoring and English usage (1) 

- more American usage, other meanings of words (2) 

- how to teach adults in particular (1) 

- understanding body language (1) 

- learning from experienced teachers about how they teach (1). 
The learning materials such as worksheets, handouts, and ^the 

"survival kit" in the training program were almost unanimously 

rated as helping the tutors "very much," with only two tutors 

rating them as "somewhat" helpful. Reasons given for the 

helpfulness of these materials highlighted the facts that they 

could review lessons at home, they could use them with their 

tutees, they were particularly good for new vocabulary and speaking 

and listening, they were good for adults, and they received quick 

feedback from the teacher. when asked what about the learning 

materials could have made them more useful, about half of the 

respondents indicated that they couldn't think of anything. Others 

(one person each) found that when using them with their TAG tutees, 

some found them too difficult, and that they wanted materials with 

more homework assignments, pronunciation practice, short stories 

with vocabulary exercises, and more writing practice 

Q 0 
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The Tutoring Process and Materials 

We asked the tutors a number of questions about their TAG 
students' work in the tutoring process. All but two agreed that 
their students come to sessions every time and do their homework. 
A slight majority rated their students as working "somewhat" hard 
to learn English, although most of the rest indicated their 
students worked "very hard, • and only one indicated the students 
worked "a little." Thus, in general, the tutors perceive the TAG 
students as fairly well-motivated. A few tutors mentioned, 
however, that the main reasons some students work less than "very 
hard" at their English lessons is because they are extremely busy 
with their outside jobs, taking care of their children, or both, 
squeezing in what they can of tutoring sessions and related 
practice. 

Particular features which the tutors think contribute most to 
their successes in teaching their students English included making 
special materials needed, the practice speaking in English, 
providing writing practice, asking the students what they want to 
learn and trying to provide that, requiring or encouraging the 
tutees to ask questions in English, and emphasizing grammar, 
vocabulary development, and, in one case, using maps to teach 
English . 

The tutors' opinions about how difficult the students found 
the videotapes, the core of the learning tools, were almost evenly 
balanced among the tapes being "too easy" (for 8 students), "just 

33 
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right" (7) , and "too hard" (5) . 4 Also, while five tutors thought 
the students found the tapes helpful for their students to learn 
English "a lot," the majority (10) thought the • tapes helped 
students' English learning only "a little." These are by and large 
the ones who rated the tapes as too easy or too hard above. The 
tapes were, however, more often than not perceived as generally 
"interesting" rather than "too long, M and were always available to 
students . 

When asked what could be better about the tapes , some 
responded "nothing, but others indicated there was the need to 
watch the tapes multiple times to benefit from them 5 , the need for 
a book to go with the tapes 6 , and a need for more full sentences 
and longer sentences. Still others thought the tapes were too 
basic, needed more conversation, needed phonetics lessons, were^old 
and needed to be updated, and two were at least mildly disconcerted 
by the tapes' California setting for Boston-based learners. 
Consistent with other information above, these responses seem to 
indicate that the tapes and other materials are generally targeted 
properly for the learning levels and needs of the TAG participants, 
but could be adjusted for some students at either end of the needs 
spectrum. 



Since most tutors answered this question differentially for each of their 
students, the numbers shown add up to more than 17. The total is not much more 
because some tutors did not answer the question. 

s Although this was mentioned by tutors as an area for improvement, the 
multiple viewing of tapes is actually a strength of the TAG design for the tapes' 
use, since learners can watch them as often as needed, at home, on their own 
schedules . 
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There is actually a manual that accompanies the tapes, but the TAG staff 
deliberately decided at the outset against using it. It is considered quite 
sophisticated and complex, and the staff wanted to focus the activities of these 
tutors (most of whom were new to the teaching process) more precisely. As an 
alternative, TAG staff developed, and trained the tutors to use, the printed 
worksheets to accompany the tape lessons. 0 
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In contrast to the tapes themselves; the handouts that 
accompany the videotapes were perceived as "just right" by the 
strong majority (12 of 17) of tutors interviewed, but were also 
seen as "too hard" for a few, and "too easy" for one. They thought 
the handouts helped the students' English speaking "a little" about 
as often as "a lot." The length of the materials was viewed as 
"just right" or "too short," each about equally. The Chinese 
instructions on the handouts were reported to be read "easily" by 
students, indicating an appropriate level of native language 
support . 

The tutors' explanations of- their reasons for the ratings of 
some inadequacies here are instructive. Among the reasons they 
mention are: some students learn easily and go through two lessons 
per session; for those with higher English levels, it's^ not 
necessary to be so repetitive, it gets boring for the students; 
some students need mostly reading and writing practice, not so much 
oral language development; and some students are too busy to use 
the worksheets (which does not reflect negatively on the materials, 
but rather the ability of otherwise busy students to used them 
well) . 

The reporting materials used by the TAG tutors included : 
Evaluation of Learner's Basic Skills, Tutor Monthly Report, and the 
Tutor-Learner Study Plan. These were rated as useful by all but 
two tutors, and all but one found them easy to use. The two who 
did not find them useful included one who prefers to report orally 
and another who is too busy to do reporting and paperwork. 

We discussed the tutors' assessments of themselves as tutors. 
Asked to provide highlights of tyhat they did best in tutoring their 
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students, the tutors responded with such items as (roughly in order 
of frequency mentioned) applications to daily life, conversational 
practice with students, grammar and vocabulary, reassuring 
students, answering all the students' questions and finding out 
more information to give them if needed, improving their own 
English in the process , teaching writing and pronunciation, and 
remembering what was difficult for themselves in learning English 
and focusing on that with the students. What they would like to 
do better includes (each mentioned by only one or two tutors) : 
making more and better practice sheets for students; generally 
gaining more experience tutoring and using English, both spoken and 
written; and teaching more writing, phonetics, grammar, and 
vocabulary , 

Particularly difficult aspects of tutoring for this group 
involved (again, mentioned by only one or two each) : being afraid, 
lacking confidence in the beginning, although this was apparently 
overcome with time; students who come late and make the tutor wait; 
students who don't do their homework; getting students to remember 
what they've learned; and keeping the same students from beginning 
to end. Two tutors indicated a problem with tutoring pairs of 
students who have very different English levels, that the slower 
one is often upset at not being able to keep up with the pace of 
the faster one and the faster one is bored. 

As a final question in the interview, we asked the tutors, now 
that they've had the training session and at least two months' 
experience tutoring, for their general suggestions to help make the 
TAG Program better. The responses were widely varied, with the 
following suggestions made by number of tutors shown after each: 
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- getting together as a group every two months or so to talk 
about their experience's and learn from each other (4) 

- more tutor training practice in conversation, to expand their 
vocabulary and speaking ability (3) 

- tutor training longer than the current eight weeks (3) 

- visit classes and talk to experienced teachers to get more 
teaching "tips* (2) 

- more secure system for getting paid as planned (some students 
apparently want to pay less) (2) 

- materials and techniques for teaching writing (2) 

- more actual practice tutoring in the training sessions (1) 

- teacher/student role playing during the training sessions (1) 

- visiting classes to see how teachers teach (and thus learn 
some additional techniques that might be useful in tutoring) 
(1) 

- make students come on time (1) 

- English/Chinese dictionaries for students (1) 

- meet with students more often than biweekly (1) 

- materials and techniques for teaching grammar and phonetics 
(1) 

- matching the same levels of students with each tutor (1) . 
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Participants' Evaluation of the Tutoring Process and Materials 

The TAG tutees generally rate their tutors very well. Over 
half the participants interviewed rated their tutors with the most 
positive descriptor we provided: My tutor explains so that I 
understand a.'l the English on the tapes and worksheets, and also 
answers my questions about other usage as well. A third selected 
another positive descriptor for their tutors: My tutor explains all 
the English on the tapes and worksheets so that I understand better 
and can use what I learn. A small portion of the TAG participants, 
three of the 36, rated their tutors less positively with: My tutor 
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goes over the lessons on the tapes and worksheets, but I leave not 
understanding as much as I would like. 

The items in Table IV- 1 were drawn from items the tutors had 
indicated were the most important things they taught. The TAG 
tutees were asked in which ones their tutor was especially helpful. 
The frequency of responses is shown for each item. As the 
responses indicate, most of the items the tutors valued came 
through to the tutees. The one exception may be that of learning 
about living in the U.S., which was indicated as especially helpful 
by only 20, still over half, of the 36. 



Table IV - 1 



TUTORS ESPECIALLY HELPFUL IN: 


NUMBER OF 
RESPONSES^ 
(out of 36) 


Explaining English words or phrases 


35 


Talking over the skit on the tapes 


32 


Helping you with problems you have learning 


32 


Making you want to work harder at learning 


30 


Learning about living in the U.S. 


20 


Other: learning how to read, pronounce words 
lots of examples to explain a new word 
newspaper articles 
dictation of new words 


1 
1 
1 
1 



The TAG tutees were also asked what activities they and the 
tutors could do in the sessions to help them learn English better. 
Curiously, some of the most frequent answers are very similar to 
those items rated as the most helpful things the tutors do. 
Nevertheless , the following items appeared. 

3S 
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Table IV - 2 



ACTIVITIES TO HELP LEARN ENGLISH BETTER 


NUMBER OF 
KESPONSES 
(out of 36) 


Have more English conversation 


23 


Learn more what English words and phrases mean 


20 


Work on pronouncing English words and phrases 
more 


19 


Go over worksheets more 


11 


Other: - grammar and phonetics 

- how to write in English 

- teach citizenship materials 

- more everyday usage 

- more examples to explain words 

- field trips to learn from "real life" 

- materials beyond the video lessons 

- have more frequent tutoring sessions 

- nothing could be better or can't think 
of anything 


4 
2 
1 
1 
1 
1 
1 
1 

3 
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The majority of the tutees rate the videotapes as "just right" 
(27 of the 36), although four found 'them "too hard" and four "too 
easy." The tapes also were viewed as helping them to speak English 
"a little" somewhat more than "a lot" (20 versus 13, respectively), 
although they were "interesting" rather than "too long" for 31 of 
the respondents. Less than half (40 percent), though still a 
substantial number, indicated that some things could be better 
about the tapes to help them learn English better. These included: 
phonetics, especially in the first few lessons (5 respondents), 
more conversations about daily life (3), longer lessons and 
dialogues (2) , subtitles for everything the teacher says on the 
tapes (2) , (1 each of the following) more examples to explain every 
word, a slower speed, update the skits, more exercises, teach 
citizenship, and English lessons on sewing (from a seamstress). 
Thus, in general, it appears that the TAG participants find the 
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videotapes adequate to help them learn English, with some fine- 
tuning suggested for improvements. 

The TAG students' assessments of the worksheets that accompany 
the tapes almost perfectly paralleled their assessments of the 
tapes themselves, as described above. That is, they were generally 
"just right," with a few indicating they were "too easy" or "too 
hard"; they helped the participants learn English "a little" more 
than "a lot"; and they were "interesting" rather than "too 
repetitive." One exception was that 13 of the 36 thought there 
were "too few" worksheets. All reported being able to read the 
Chinese instructions on the sheets, and all but two people found 
them helpful (two who indicated they could do the lessons without 
the Chinese instructions) . 

It was noted above that some of the tutors like to mak^ up 
their own extra worksheets and provide other supplementary 
materials and activities. This was verified by 15 of the 35 
tutees ' reports that their tutors do other learning activities 
beyond just the tapes and accompanying worksheets. They report 
being taken on field trips to learn how to shop and read prices (3 
TAG students), extra phonetic exercises (2), practicing 
conversation on pictures the tutors bring (2), grammar exercises 
(2) , (one mention each for the following) watching TV, homework 
from another ESL program, asking a lot of questions, reading 
newspapers, writing short essays about family and holidays, extra 
exercise sheets, and discussions about other topics relevant to 
daily survival and adjustment in the U.S. 
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Summary 

In the tutors' views, the tutor training program was effective 
in making them feel prepared to tutor. They highlighted the 
achievement of the two main program objectives: better English 
pronunciation and teaching skills. While the best features of the 
training program were seen as learning good teaching skills, having 
a good model in their trainer, ways of motivating students, and 
speaking with more experienced tutors, they also suggested that 
improvements might be made in having more time to learn, building 
more confidence, learning more American usages of words, doing more 
reading, and learning techniques particularly useful for adult 
students. There was also a repeated theme of desiring that more 
tutoring practice be part of the training prqgram. The materials 
used in the tutor training were unanimously seen as very helpful, 
although a few found the reporting materials too hard to use. 

The tutees generally rate their tutors very well, with only a 
few commenting that they don ' t - understand all they would like to 
from the sessions. After some experience tutoring, some tutors 
indicated the need for help in making more and better practice 
sheets for their students. Some tutors apparently do this to a 
great degree and others do not, a feature mentioned by both tutors 
and tutees. It was also suggested that periodic — say, bimonthly 
sessions be held for the tutors to come together to discuss 
their ongoing experiences tutoring. 

The tutees almost unanimously rated the videotapes and 
supportive materials as H just right" in level of difficulty for the 
TAG participants to learn English, with a few indicating they were 
either "too easy" or "too hard." The tutors, however, were almost 
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equally split as to whether the tapes were "too easy," "just 
right," or "too hard." Some areas for improvement noted included 
having longer dialogues and lessons, more subtitles for the 
teachers' speech on the tapes, books to accompany the tapes, more 
conversations about daily life, slower speed, and updating the 
skits . 
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V. EVALUATION OF PARTICIPANTS' ENGLISH LEARNING 



The TAG Program's second goal was to teach English to the 
participants on the waiting list for AESL classes in order to 
improve their English skills adequately for their entry into the 
Level 2 classes. We also observed the AESL classes of the three 
Level 2 AESL teachers with TAG "graduates" in their classes to 
assess the TAG participants' uses of English and other indicators 
of learning skills compared to those of the other students, and 
tested all students in both classes with the BEST at the beginning 
and the end of the class terms 7 . 

English Background and Progression 

Almost all of the participants (27 of 35) indicated that they 
had studied English some before the TAG Program. All but one of 
those 27, however, studied it for less than six months, generally 
through classes at other community-based organizations or through 
churches. Three indicated that they had studied at least some 
English in school in their native countries, though it was 
generally in their youth and they were now adults, and thus they 
attained little proficiency and remembered little. Asked if they 
used any English words or phrases when they first came to the U.S., 
13 of 35 (37 percent) answered that they did, although the majority 
(63 percent) indicated that they did not. Of those who did, the 
most frequently used words and phrases included basic greetings 
such as: Hello, Thank You, Excuse me, Good morning, How are you?, 



7 The BEST was adm i n i stored by tho TAG^sc^ff, after training by DAC staff. 
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and so on. Other examples given, by one or two persons each, 
included: "Can you give me ...?," "Where is the bathroom?," "I'm 
sorry, " "My name is ... , " "My telephone number is . . . , M "Yes" and 
"No," and "chicken wings." 

To assess their confidence in using these elements of English 
upon arrival, we asked whether it was easy or hard to use these 
words and phrases, whether they used them sometimes or often, and 
with whom. Of the 11 responding, seven found them easy to use and 
six found them hard. Five used the phrases often compared to six 
who reported using them only sometimes. The people with whom they 
spoke this basic English included primarily their neighbors and co- 
workers, though less frequently new people they would meet. 

After they had been in the U.S. for awhile, however, but 
before the TAG tutoring, almost two-thirds (23 of the 36) reported 
having learned more English words and phrases and trying to use 
them, though 13 indicated they did not learn any more. For those 
who reported that they did, the examples of the types of words and 
phrases they used showed noticeably greater sophistication: 
Greetings were still predominant, though extended in content; Do 
you have a job?, What is your name?, Where do you live?, elevator 
use instructions, days of the week and months of the year, job- 
related vocabulary, and one person indicated "verbs and third 
person singular." Despite the increased sophistication, however, 
their confidence did not appear to increase as yet. Ten found 
these words and phrases easy to use and 11 found them hard; 16 
indicated they tried to use them sometimes, while only two tried 
often . 

44 
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After having participated in TAG tutoring for some time (at 
least two months for all those interviewed) , 34 of 35 indicated 
they had learned new words and phrases. These words and phrases 
are generally expansions of the same themes about which they spoke 
above, including, as would be expected, considerably more detail in 
phrases and in the now longer and more complete sentences and in 
additional vocabulary. The main themes cited by interview 
respondents were: housing and apartment descriptions and 
procurement process, local transportation, shopping interactions, 
and greetings for casual conversation. Moreover, their confidence 
appeared to have increased considerably by this point, with 23 of 
32 indicating it was now easy to use this English and only seven 
finding it hard (plus two who indicated a mix, "sometimes easy, 
sometimes hard") . Twenty-two use their rapidly developing English 
sometimes, whereas 10 report using it often. The majority (13 
citations) use English mainly in their job situations, either with 
co-workers or with clients and customers. Next, they use it when 
shopping, then with "strangers" (presumably meaning when they meet 
new people) . 

Motivation for Learning English 

Along with the changes in English usages described above, we 
also attempted to examine the participants' main motivations for 
wanting to learn English. We asked them to indicate Yes or No to 
a list we presented as to whether or not each item was a reason 
they wanted to learn English. The results, with the number of 
participants responding, are shown in Table V-l. As indicated 
there, shopping and job-related needs that is, economic survival 
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or growth through commerce and employment are the most 

motivating for the participants, cited by just about every 
respondent. The other reasons appear to motivate substantial 
numbers of participants, but don't appear to be as universal as the 
first two. 

Table V - 1 



REASONS FOR WANTING TO LEARN ENGLISH 


(out of 36) 


talking to clerks at shops 


34 


your job or getting a job 


33 


taking the T or taxi or bus 


27 


going to school in the U.S. 


26 


going to the doctor or emergency room 


21 


other (included): 

- general communication with 
Americans 

- citizenship test 

- survival 

- travel/find directions 

- respect for speaking well 

- read mail 


6 
5 
4 
2 
1 

1 


talking to your neighbors 


17 


talking to your children's teachers 


16 



From the interviews with the sample of tutors, we also 
ascertained their perceptions of the participants' motivations for 
learning English. They largely reinforced the same reasons cited 
by the participants, indicating, in rough order, that job-related 
needs, shopping, transportation, medical care, and daily living in 
general represented the areas of greatest need for the TAG 
participants to learn English. The AESL teachers interviewed also 
concurred that job-related needs for English was a strong 
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motivator, with shopping, medical care, transportation and travel, 
and citizenship preparation being the next highest priorities. 

Another motivation indicator asked of the AESL teachers was 
how hard the TAG students worked in class, with the response being 
generally "very hard, " with the exception of one student who was 
very busy with a restaurant business. For those students without 
TAG backgrounds, one teacher reported that there was a wide range 
in how hard they worked at learning English through the class and 
one reported that the majority (about 80%) worked hard also, but 
that about 15% only worked "somewhat" and about 5% worked "a 
little." 

Ways of Learning English 

We also presumed that some information about the participants' 
English skills and learning might be inferred from their degree of 
difficulty or ease in using the videotapes and supplementary 
worksheets. Thus, when asking about these items we found that 27 
(of 35) found the videotapes "just right," four found them "too 
hard," and four found them "too easy." Their assessment, however, 
is that the tapes helped them learn to speak English "a little" (20 
respondents) more than "a lot" (13 respondents) . Parallel 
information concerning the accompanying worksheets indicated 
essentially the same results: 22 found them "just right," seven 
found them "too hard," and for six they were "too easy." The 
worksheets helped 24 participants speak English "a little" and 
helped 11 speak it "a lot." 

These findings are not inconsistent with the intended 
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structure of the program. That is, not all learners are supposed 
to find the videotapes "just right" for learning English easily. 
If that were the case, there would be no need for the tutors or any 
of their judgments about supplementary activities tutees might 
need. The TAG Program was not structured as a stand-alone video 
course. Rather the videotapes are a focal medium for the more 
complex learning process, which is facilitated by the tutors and 
the curriculum they implement around the tapes. 

Finally, we asked about other ways in which the respondents 
were learning English. Watching ..TV and movies and talking to their 
children were each cited by 25 of the 36; 18 and 22 respectively 
learned from work and from reading signs, packages, and work 
instructions; fewer numbers (ten or less of each) learned from 
talking to neighbors, family members (other than their children), 
newspapers, a radio program that teaches English to Chinese people, 
an electronic dictionary, and books. 

AESL Classroom Performance 

Description of AESL Classes. Our intent in observing the 
Level 2 AESL classes was to get an idea of what TAG "graduates 11 
would be experiencing in a classroom with non-TAG students. We 
looked at the range of activities, the physical environment, the 
use of English and Chinese language and bilingual instruction, and 
the materials used by the teachers and students. 

We also rated the students on their level of spoken English. 
This rating in the context of one observation likely cannot reflect 
reliably the speakers' abilities, since the topic may be more or 
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less familiar, the presence of an outsider may affect the students, 
or the activities observed may not demand much independent verbal 
response (e.g., silent reading or whole class recitation). 

Range of Activities. Activities typically ranged from the 
teacher introducing a new idea such as the verb "can" to phonetics 
and writing examples on the board, to playing a cassette tape with 
a dialogue, to students taking turns answering questions or 
reciting the content of the lesson for pronunciation and syntax 
practice. Students would also work silently either reading a 
passage or writing answers in their workbooks. Paired activities 
were a^so conducted where students were supposed to discuss a topic 
with new vocabulary or a handout with pictures. The English taught 
in this adult program is primarily "survival English," i.e., 
vocabulary and situations in which a student would experience in an 
everyday setting as opposed to an academic setting. 

Physical Environment. Classes were held in either a classroom 
used during the day by a children's class, a large classroom near 
the QSCC administrative offices or on the third floor in the Oak 
Street building. 

The children's classroom is very spacious, but the desks and 
chairs are small. It is well-lit with florescent lighting, but 
there are no windows. It has a wall-size blackboard and is 
decorated with historical figures from many ethnic groups as well 
as children's artwork and writing samples. Usually there was 
little noise, however, occasionally street noises such as sirens 
and horns honking and recitation fro.T. the class next door (only 
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divided with cubicle partitions, not walls) interrupted the quiet 
surroundings . 

The large classroom is also spacious and f lorescent-lit with 
two walls covered with blackboards . There are several large tables 
with folding chairs. This room is decorated with a student-made 
quilt, some English expressions, and the alphabet at the top of one 
of the blackboards, but is otherwise plain- It was very quiet as 
it is surrounded by other rooms. 

The Oak Street classrooms are small with new, bright paint and 
many decorations like posters t maps, and magazine pictures. 
Several tables are arranged in different ways depending on where 
the blackboard is located in the particular classroom. The 
lighting is good, coming mostly from the many" windows during the 
day and the florescent lights in the evening. These rooms are 
comfortable and homey-feeling, but hot in summer and noisy from 
traffic and construction in the street and children playing at the 
daycare downstairs. 

English and Native Language Use. Most of the students speak 
Cantonese and/or Mandarin. Some of them speak Vietnamese, 
Toisanese, and some other dialects. English was used quite a bit 
by the teachers , who are all bilingual , but was also translated 
either right away or later if the student didn't understand 
readily. Sometimes the teacher would introduce the new material 
entirely in English and ask questions in English, too. They often 
explained further or verified the students' comprehension in 
Chinese. The students often responded in English, but if they were 
not sure of what to say would ask for the word in Chinese, the 
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teacher would answer in English, and the student repeated in 
English . 

Materials. In addition to the blackboards, which were used to 
introduce new ideas, students' examples, or other types of language 
teaching, materials used by the students and teachers varied. 
Usually, students referred to their QSCC-made text and workbooks 
and wrote answers directly into them. These materials have Chinese 
translations for much of the instructions and new vocabulary. Also 
used a great deal were handouts with stories, questions, or 
pictures to stimulate discussion „ among students. Occasionally, 
other materials were used such as a pair of shoes to imagine the 
person who wears them or a cassette tape and map handouts to listen 
to a dialogue for new vocabulary about giving directions. 

Impressions. We observed in the several classes of about 15 
to 20 adult students, led by three different teachers, a positive, 
comfortable learning environment. The students were encouraged but 
not forced to participate in class and the teachers were helpful 
and adapted their attitude depending on the student (s) being 
addressed at the moment . Students volunteered to respond or have 
conversations on a range from some very willing to most willing to 
a few unwilling. 

The students apparently got on well with each other and the 
teachers, chatting and joking occasionally in Chinese. The only 
time this practice seemed to interfere with the learning process 
was during paired activities when students were instructed to have 
conversations in English, using the handouts for example. Students 
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were reluctant to do this unless the teacher was next to their 
desks helping direct their activity. 

Finally, there was no clear difference between TAG and non-TAG 
students in the AESL classes. However, any conclusions from this 
observation would be unwarranted, because the proportions of TAG to 
non-TAG differed from class to class, as well as the 
characteristics of the students, who had been their tutors, what 
other English-learning practices they participated in, etc. 

Student English Performance. We assessed the TAG students' 
performance in using English from the interviews with the AESL 
teachers, from the students' test scores, and from observations of 
their speaking in class. At the outset of the classes, the 
teachers indicated that the TAG students responded in class at a 
variety of levels two or three "volunteered to speak at any 
opportunity without hesitation, " another two or three "spoke when 
requested or during recitation, " though the majority "almost never 
spoke." The non-TAG students in the Level 2 classes, by 
comparison, were more homogeneous and mid-level in their responses, 
generally speaking "when requested or during recitation" to "often, 
with some hesi tat ion . M Considering other information available, it 
appears that the fact ".hat the non-TAG students, most of whom went 
through a Level 1 AESL class, have learned an appropriate and 
consistent level of response which becomes fairly standard upon 
entering a Level 2 class. The TAG graduates, on the other hand, 
have not been in a standard classroom environment, at least 
recently, and thus respond quite differently when placed in that 
situation . 
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At that point of the interview, about two-thirds of the way 
through the class term, the teachers indicated that many of the TAG 
students did improve in their classroom responses, with about half 
responding in English without hesitation, or with only some 
hesitation. The non-TAG students were assessed as having stayed at 
about the same levels of responsiveness as they were in the 
beginning. Thus, it appears that the TAG students "catch up" with 
the others in terms of fitting into the general classroom response 
styles . 

Other differences between the TAG and non-TAG students that 
the AESL teachers pointed out based on their experiences in 
teaching both groups (this being the first cohort of TAG graduates 
placed in AESL classes) included: 

a) TAG students appear to have fairly high literacy skills, 
both in their native and English languages, but their speaking 
skills are lower than those of the non-TAG students; 

b) the non-TAG students more often expect choral recitation 
type activities in class, whereas the TAG students do not; the 
TAG students appear to try to figure things out for 
themselves, work individually better, and follow instructions 
better on their own. 

Students in both classes were tested with the BEST in their 
third week (pretest) and 22nd week (posttest) of the first 24-week 
term of 1993 . The scores and gains for the oral and literacy 
subtests were compared from pretest to posttest for the TAG and 
non-TAG students, with the results shown in Table V-2. The 
students enrolled in second term classes, beginning in July 1993, 
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were also tested with the BEST during the third week, and those 
pretest scores are also shown in the table. 8 



Table IV-2 

BEST Scores for AESL Level 2 Classes — 1993 9 



Term 


Subtest 


ORAL 


LITERACY 


Group 


TAG 


non-TAG 


TAG 


non-TAG 


First 


N 


11 


17 


11 


17 


Pretest 
Mean 
(s.d.l 


33 .1 
(13.0) 


29.4 
(15.7) 


36.7 
(9.8) 


31. 6 10 
(11.6) 


Postest 
Mean 11 
(s.d.) 


39.7 
(12.5) 


42 .6 
(16.9) 


42.3 
(7.9) 


38.1 
(10.2) 


Mean 
Gain 


6.6 


13 .2 


■ 5.5 


6.5 


Second 


N 


17 


12 


17 


12 


Pretest 
Mean 
(s.d.) 


22 .1 
(12.4) 


33.3 
(16.9) 


26.1 
(9.8) 


32.8 
(12 .8) 



As the test data show, the TAG students began the classes in 
the first term with slightly higher oral and literacy skil.'.s, as 
measured by the BEST, than did the non-TAG student -. . his 
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8 Since the evaluation ended in September, only the pretest scores for tiiis 
term have been gathered and reported herein. 

9 Form B was used for the pretest, Form C for the postest, and scores for Form 
C were converted to scaled scores for comparison with Form B (in which raw scores 
and scaled scores are equivalent). 

10 The testers reported that some non-TAG students did not finish the last 
part of the literacy subtest, thus deflating their scores somewhat. 

11 One of the two teachers indicated to the students that this test would be 
used as the course final exam. This condition was different from that of the other 
class 
most 



=>d as the course tmal exam. This condition was ainerent trom lucjl ul cnu u 
iss. Its effect, if any, would likely have been to slightly inflate the sco 
stly for the non-TAG students who made up the majority of this class. 
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measured by the BEST, than did the non-TAG students. This 
occurrence may be significant in that it suggests that the TAG 
Program was effective not only in bringing participants up to the 
same levels as those who went through AESL Level 1 classes, but 
possibly even higher levels of oral and literacy English skills. 
In terms of gains during the class, the TAG students gained 
somewhat in oral skills, but only about half of what the non-TAG 
students did. In literacy skills, however, both groups gained 
about the same amount, thus maintaining the TAG students' slightly 
higher levels of literacy skills. (None of the gains are 
statistically significant, however, and the number of students with 
both pre- and posttests is small. As a result, conclusions about 
these data should be interpreted with caution.)' 

What these scores suggest is that the TAG Program has prepared 
the students very well with the oral and literacy English skills to 
enter a Level 2 AESL class. The data also suggest that the program 
has given them the background to at least maintain progress in the 
class at the same level for literacy skills as those who have gone 
into Level 2 from a Level 1 class. The TAG students in Level 2 
classes have not gained in oral skills, however, as much as the 
other Level 2 students did. 

The second term scores indicate that the TAG students begin 
their AESL Level 2 classes with considerably less proficiency in 
both oral and written English than do their non-TAG classmates, and 
less than the TAG students of six months back, in the first term. 
The TAG students scored about 11 points lower than the non-TAG 
students in oral English and over 6 points lower in literacy. 
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Similarly, they were about 10 to 11 points lower in both subtests 
than the TAG* students pretested in the first term. 

The non-TAG students in these second term classes scored about 
the same as their non-TAG counterparts at the beginning of the 
first term, suggesting that there was nothing in the testing 
procedures from the first to the second term that accounts for the 
change, but rather something in the demonstrated proficiency of the 
TAG students. They may be less able learners, have weaker English 
backgrounds upon TAG entry, have not been tutored as well as the 
TAG students who entered classes the first term, or may simply not 
be as strong performers in a formal test as the non-TAG students 
are. This last possibility is a good one for at least two reasons. 
First, the spoken English observation data (detailed in Appendix B) 
indicates that TAG and non-TAG students performed about equally 
well in the beginning of term 2, and at about the same levels they 
did in the beginning of term 1. Second, the interviews with the 
AESL teachers (before this test data was analyzed) indicated no 
special problems with the TAG students in their classroom 
performance. Thus, despite the curiously low test scores, there 
appears to be no threat to the TAG students' successful functioning 
in AESL Level 2 during the second term. 

Student English speaking was also assessed by observers at the 
beginning, midpoint, and end of each class's term. Each time a 
student spoke, he or she was rated on a 4-point scale 12 as follows: 

5G 



2 Adapted by the authors from the BEST pronunciat ion scale . 

TAG Evaluation Pinal Report - 52 



0 = Speech is almost always unintelligible. 

1 = Speech is frequently incomprehensible. 

2 = Speech is generally understandable, but occasionally 

difficult or impossible to comprehend. 

3 = Speech is readily understandable. 

The details of the instrument's reliability and validity, as 
well as all data, are presented in Appendix B. A summary of the 
findings is that class mean ratings ranged from about 1.2 (at the 
beginning of the terms) to 2.3 (near the end of the term) on the 
scale above, that the non-TAG students were almost always rated 
very slightly, though insignificantly, higher on average than the 
TAG students (the observers did not know who belonged to which 
group until after the class was observed) , and that the TAG 
students gained at the same rate as the non-TAG students. 

Summary 

The immigrants enrolled in the TAG Program generally had some 
minimal training in English before their U.S. arrival, and a few 
months of English study in the U.S. before participating in the TAG 
Program. They report being able to use very little English upon 
arrival, and they made little progress in English usage before TAG. 
After two or more months of TAG tutoring, however, they report a 
substantial increase in their English vocabulary, speaking with 
more detailed phrases, and using longer and more complete 
sentences . In addition, their confidence in using English is 
reportedly increased substantially by this point, even after little 
increase in earlier learning stages. 
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The main motivations for learning English are economic to 
get an initial or better job and to conduct their shopping. Other 
uses of English are present but secondary. Moreover, the TAG 
students in AESL classes are reported by their teachers to work 
somewhat harder than other students, possibly indicating higher 
levels of motivation . 

The majority of TAG participants found that the English levels 
used in the main tutoring tool, the videotapes and supporting 
materials, were "just right" to help them learn well, with a few- 
finding them either "too difficult" or "too easy." In addition to 
these formal course materials, however, other television programs 
and movies were widely reported to be of additional assistance in 
learning English, along with talking to their children who were 
learning English in school, features which the TAG Program might 
attempt to capitalize on in future developments. 

Once the TAG students enter AESL Level 2 classrooms, they 
appear able to perform well. Interviews with the teachers 
indicated that, while their other students generally respond in 
class at a mid-level of confidence and ability, the TAG students 
varied widely, from speaking without hesitation at any opportunity 
to almost never attempting to speak. After a few months in the 
classes, however, the TAG students appear to "catch up" in response 
level to the norm. 

As judged by standardized English test scores, the TAG 
students gained about as much in English literacy as the other 
students. The non-TAG students, however, showed gains in oral 
English skills at about twice the level of the TAG students. This 
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latter test result would be worth examining further- Using the 
outside observers' ratings of English performance in the classrooms 
as the criterion, however, the TAG students performed only very 
slightly lower in English level than the non-TAG students, and both 
groups gained at about the same rates. 
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VI. ESTABLISHMENT OF AN EVALUATION SYSTEM FOR TAG 



The third goal of DAC's evaluation effort was to establish an 
overall evaluation system for the TAG Program and to help develop 
TAG staff capability to implement it. In this section, we present 
more detail about that system through an application of it, along 
with the activities undertaken this past year to develop the TAG 
staff's capability to carry it out on their own. 

Ongoing Program Analysis Framework 

This system is not only intended to include the development of 
measures of program outcomes, but also to include means by which 
periodic, systematic feedback on program progress can be provided 
for ongoing program adjustments and development. The framework for 
accomplishing this goal is that of the Discrepancy Evaluation Model 
(DEM), introduced early in this report. The specific application 
of the DEM framework to the TAG Program resulted in the model 
presented earlier in Figure 2. 

Many of the outcomes from the DEM application to TAG have 
already been treated throughout the earlier parts of the report, 
for they were used as a starting point in deciding what to measure 
in this evaluation. Others among the outcomes, however, are 
program process features or are items for which there was no 
feasible measure available. As a result, we discussed many of the 
elements of the DEM with the TAG staff near the end of the 
evaluation project. In the Figure 3, we present the status of each 
in terms of its attainment and reference evidence of that 
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attainment where it was presented earlier. This set of results, 
however, is not meant to be a checklist for tallying successes and 
failures, but rather a heuristic for discussing some strengths of 
the program and areas for attention. The best use of the DEM 
application is through regular staff internal analysis. 

A few program areas for attention are suggested through the 
analysis above, in some cases in combination with other findings 
from earlier sections. First are those outcomes having to do with 
program elements of tutor competence and quality (1C and 2E) . 
While the interviews with TAG participants did indicate that they 
were generally satisfied with the tutoring, that evidence might 
well be influenced by "halo effect." Specific competencies in the 
three main objectives of the tutor training — English; tutoring, 
and reporting skills might be specified and measured so that the 
tutors are certified before they begin tutoring. Then the program 
component of monitoring tutor quality (2E) , providing more formal 
supervision and feedback, and improving the TAG tutoring process 
could be more effectively carried out. Information from this 
monitoring might then contribute to the ongoing revision of the 
tutor training curriculum and materials (IB) . 

Second, there was also no evidence that the desired ongoing 
revision of the TAG materials (2A) took place, and there were 
concerns about the appropriateness of the videotapes and materials 
in most effectively teaching English to the TAG participants (also 
2A) . Unless another method is used, however, these tapes were the 
most appropriate available on the market. It was noted in section 
IV cf the report that a good number of tutors desired occasional 
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meetings with TAG staff and other tutors to discuss problems, share 
techniques they've found helpful, learn new methods, and adapt or 
develop their own materials. If such meetings did take place, they 
could serve as the primary vehicle for improvements in this area, 
as well as providing still more information for the continuing 
improvement of the tutor training program. 

From the AESL teacher interviews, two suggestions were made 
for additional areas to consider in improving the program, areas 
akin to those above. The first stems from the observation that TAG 
"graduates" in the Level 2 classes are used to working individually 
or in pairs with their tutors and are used to learning through 
following steps in written or audiovisual materials. When they get 
into a classroom, many appear not to be used to, or confident 
about, interacting in larger groups. They seem to need help with 
the transition to overcome their reticence and thus benefit well 
from a large class learning setting. Suggestions included: a) 
building more interaction with oral English language activities 
into the tutoring process, at least as the time to enter an AESL 
class approaches; and b) in AESL classes with TAG ^graduates, H 
guide the teachers to conduct small group activities first so that 
the TAG members get used to speaking up in groups, then move into 
large group activities. 

The second suggestion addresses an issue that arose 
periodically in sections IV and V of this report, from both the 
tutors and tutees namely the difficulty of tutoring pairs of 
students in which each member is at a very different English 
ability level. One obvious solution is to develop a better system 
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to pair tutees with similar abilities, such as a master matrix of 
students' levels and schedules and tutor schedules to see if better 
combinations emerge, recognizing that there may be no possibility 
of getting every pair perfectly matched. Another potential 
solution may be to consider peer tutoring within the pair, under 
the guidance of the formal TAG tutor and/or TAG staff. Such a 
solution would involve special training or supportive assistance 
for the tutor in such a situation. The TAG staff is well-aware of 
this problem and has devoted much apparent energy to its solution. 
The extent to which more special solutions are sought depend, at 
least partially, on the extent of the problem. If it is not known 
how many tutee pairs are thus mismatched, perhaps that could be 
examined more carefully. 

TAG Staff Evaluation Capacity Building 

Progress has been made in building TAG staff capacity to 
implement their own evaluation. First, the TAG staff worked 
through the development of the DEM application with DAC staff. 
This exercise provided an experience of program analysis in a 
manner which both laid the groundwork for evaluative judgments to 
be made and which fits into the ongoing program management, 
development, and decision-making process. 

Second, the TAG staff reviewed, critiqued, and suggested 
revisions to the interview protocols drafted by DAC. In doing so, 
V gained experience in interview development methods. Once the 
drafts of the interview protocols were established, TAG staff 
ained by DAC in conducting the interviews with TAG 
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VII. CONCLUSIONS AND DISCUSSION 



Based on the information gathered from all sources, the 
following conclusions about the effectiveness of the TAG Program 
are suggested. 

A. The TAG Program is clearly an efficient opportunity for 
immigrants to start formally studying English sooner than available 
classroom opportunities allow. This is especially true for those 
who fit into demographic groups with the greatest scheduling and 
other logistical barriers. 

The TAG students are generally slightly older, less educated, 

and more often married and have children than other immigrants who 

have contact with QSCC's English programs. This group may find it 

easier to begin their study of English in a less formal way than a 

traditional class and may also find the scheduling easier than the 

classes would demand. Since they are generally less educated, they 

may also find the tutoring and videotapes a less intimidating 

transition into more formal English study than classes. While this 

characterization holds in general, the TAG participants do in fact 

represent the entire range of demographic characteristics, and thus 

TAG is not only limited to a select group. 

B. The tutor training appears to be successful in developing 
English and tutoring skills to at least an adequate level for 
effective tutoring. 

This success is evidenced by the tutors' unanimous reports 

that they were prepared to tutor well by the end of the training, 

their tutees generally rated their effectiveness very well, and the 

TAG "graduates" in the Level 2 AESL classes are generally 
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performing at least adequately. Additionally, although some 
problems with the tutoring process were cited, such as some tutors 
lacking full confidence in their abilities, these problems did not 
appear to be systemic. Rather, they seemed relatively minor and 
were overcome individually or with the supporc of the TAG staff. 
For this problem in particular, nervousness or lack of confidence 
is generally characteristic of all new teachers. Moreover, this 
lack of confidence was recognized by the program staff as a problem 
to overcome, and the program was designed to build the tutors' 
confidence as part of their, skills. In light of this 
understanding, that only very few of the tutors interviewed cited 
nervousness or a lack of confidence may be taken as a positive sign 
of the TAG Program's impact. (On the other hand, there may have 
been some cultural press against admitting to nervousness or lack 
of confidence, and thus more tutors may have been more insecure 
than they admitted to the Caucasian interviewers.) 

Some of the suggestions for improvement noted by both tutors 
and tutees, described in more detail in sections IV and V, might 
well be taken under consideration by the TAG staff for program 
development opportunities. Highlights of these included having 
more tutoring practice in the training sessions, making 
supplementary materials and exercises based on assessing tutees' 
needs, and occasionally meeting with other tutors and TAG staff to 
discuss problems. 
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C. The videotapes and supportive written materials appear to be at 
the appropriate level of challenge for most of the tutees, and the 
tutors appear to use them well in teaching English* 

Some of the tutees, however, reported that they found the 
tapes and materials too easy or too hard. As discussed earlier, 
that is not a problem in itself, for the TAG Program design 
intended to use the videotapes and materials as a core around which 
the tutors would work with learners. The tutors have as one of 
their roles to adapt the learning activities to the abilities and 
speeds at which the learners can use the tapes and other materials. 
The attention to this matter signalled by some tutors' comments, 
however, indicates that the issue is a concern for them. TAG 
staff's response might include a clarification and reemphasis on 
the tutors' needs to adapt to varied learner paces, and perhaps 
more assistance in helping them do so. This individualizing could 
be accomplished through a part of the tutor training program which 
addresses how to assess the pace and difficulty levels of tapes and 
materials, and how to modify those two as needed. If this is too 
much for an already crowded tutor training curriculum, then perhaps 
a follow-up session could be conducted for those who need it after 
a month or so of tutoring. The latter suggestion would address a 
desire expressed earlier by some tutors for periodic follow-up 
sessions, as well as to address the problems of the pair-level 
differences discussed earlier. 

Another option may be to establish a resource file available 
to all tutors. This resource file could furnish activity ideas, 
books, or other worksheets and materials provided by TAG staff and 
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other tutors. Then each tutor could obtain help in expanding his 
or her repertoire of techniques or learning activities for tutees. 



D. TAG appears to have provided a substantial boost to the 
participants ' English learning and confidence in English usage. 

From the interview data, there appeared to be slow, if any, 

progress in learning English through formal study in the native 

country and in the general acculturation process in the U.S. The 

TAG Program rapidly expands the participants repertoire of English 

vocabulary, phrases, and confidence in using those. Moreover, 

their motivation to work at learning English appears high. It 

stands to reason that the TAG Program is effective in enhancing 

that motivation by giving the participants . a tangible learning 

activity to engage in, as compared to doing no formal learning 

while on a course enrollment waiting list. 

E. TAG participants who enter AESL Level 2 classes progress at 
least as well as their non-TAG classmates in English literacy 
skills f jbut not nearly as well in oral language skills. 

This conclusion speaks well of the TAG Program in terms of the 

students' progress in classroom literacy skills, but not quite as 

well for the oral skills' component. The TAG students' gain in 

oral skills, however, was about one-half a standard deviation, and 

is thus considered to be "educationally significant." 13 Further, 

the TAG Program developers assumed early on that the TAG tutoring 
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13 Gains of one -third a standard deviation or more are considered 
■educationally significant" by the U.S. Department of Education's Program 
Effectiveness Panel (Ralph, J. and Dwyer, M.C., Making the Case: Evidence of Program 
Effectiveness in Schools and Classrooms . U.S. Department of Education, Office of 
Educational Research and Improvement, 1988) . 
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would not be as effective as a class in developing the students' 
English skills. Thus, the lower gains in oral skills are not a 
blemish on the program, but rather a statement of relative strength 
and weakness from this limited evidence. It appears worth some 
attention, however . 

Two considerations are suggested . One, the curriculum and 
teaching techniques might be examined more carefully to determine 
if there is some mismatch between the TAG preparation and the 
classroom activities with respect to oral skill development. If 
any are ident if ied, ~ then perhaps shoring up those discrepancies 
might help, either in the tutoring program or in the AESL classes, 
or both. Two, these findings are based on a relatively small 
number of' students tested and a one-time pre-posttest occurrence. 
Thus, the. finding should be accepted with caution, for it may be an 
anomaly of the small numbers and single event. 

In addition to their lower oral test performance, many TAG 
students were reported to be somewhat reticent in the classes in 
the beginning. Some overcame that reticence and began speaking up 
more often and with less hesitation over time, while others still 
held back. This is consistent with the slower progress in oral 
skills shown on the BEST data, but the cause of the problem is not 
clear. Do the students hesitate to speak up because their skills 
are inferior to the others? Or do they progress at a slower pace 
in oral skills because they don't speak up and thus take advantage 
of the oral skills learning opportunities in class? 
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F. The tutor competency, monitoring, and ongoing improvement 
component of the program could be better formalized. 

The central issues here are the lack of a monitoring system 
for tutor supervision and feedback (both from TAG staff to tutors, 
and from tutors to TAG staff) and the expressed, and generally 
desirable, need for continuous improvement of the tutoring 
materials and techniques. A monitoring or supervision system seems 
essential to the operation of any set of employees, especially such 
a loosely affiliated group as these tutors, who perform this 
function on a very^limited time basis. Given that many tutors also 
expressed a desire for occasional meetings to share techniques, 
discuss problems, learn from others, and that they try to develop 
or adapt materials to use with their students, - a monitoring process 
might be combined with periodic "inservice training" and problem- 
solving sessions for the tutors. ■ That way the monitoring and 
ongoing training could be reasonably informal and unobtrusive, yet 
contribute to the continuous improvement of the tutoring process. 
Moreover, in such periodic sessions, the practicing tutors might 
express needs or present new products or techniques that they have 
developed. These could be incorporated into future tutor training 
curricula and instruction and made available in a resource file. 

The notion of tutor competency can also be partially 
integrated with the foregoing in the following way. If a set of 
competencies for tutors were specified, then it would have at least 
two uses. One, tutors would have to be certified at the end of 
the tutor training before they could tutor, thus ensuring the 
quality of the tutors. Two, the competencies could serve as the 
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basis for ongoing monitoring; perhaps by TAG staff observation of 
tutoring sessions, or through tutor self -reports in the types of 
meetings suggested above. The monitoring need not be so formal as 
a full-time employee evaluation system might be. A set of 
competencies for tutor and TAG staff joint discussion and tutor 
self -evaluation, however, might help focus efforts better, promote 
skills that the TAG staff wants to see improved generally, or 
simply form the starting point for discussion at tutor ongoing 
training/problem-solving sessions . 

We also recognize that not all tutors could attend regular 
meetings in addition to their tutoring, other jobs, classes they 
may be taking, family responsibilities, and so on. Even if the 
sessions were voluntary, they would provide some benefit to the 
program. Perhaps some incentives, probably non-monetary, such as 
additional materials, books, or the chance to hear invited 
speakers, could be provided to encourage as high a rate of 
participation as possible. 

G. TAG staff evaluation capability has been developed, but its more 
complete establishment appears blocked by time availability or 
commitment to evaluation activities . 

The original plan for this project included additional funding 

for the TAG staff so that they would have time available to conduct 

evaluation activities and incorporate them as a normal part of 

their program functions. Significant progress in that regard has 

been made, as described in section VI above. Some of the 

activities for which the TAG staff were responsible, however, were 

very difficult to conduct in timely fashion. We recognize a number 
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of features that have been problematic. One, new staffing and 
staff assignments occurred — these often delay activity and 
produce uncertainty of individuals' roles and functions. Two, 
evaluation activities in general are difficult for program 
personnel to take on with the same legitimacy as their direct 
service activities. And three, the original plan by TAG and DAC 
did not specify how much time needed to be allotted for specific 
activities. After the experience of the past year, the TAG staff 
now has a clearer set of expectations for time and activities and 
can plan more precisely the evaluation tasks that need to be 
conducted. It is recommended that for any evaluation activities 
QSCC commits to, they be scheduled and staff responsibilities 
assigned at the onset of each class term, so that they do not get 
neglected during the pressure of events during the terms. 

One original intention of this year's grant was to end with a 
Program Effectiveness Panel submission on the program if the data 
warranted. We have concluded that the data are not extensive 
enough from this first year's evaluation to make a strong claim 
(based on a comparison with the two successful submissions we 
reviewed) . Therefore, a submission will not be prepared this year. 
Rather, TAG should attempt to build appropriate ongoing evaluation 
activities into the program so that a submission can be prepared at 
some point in the future. 

These -appropriate" evaluation activities for the PEP purposes 
should include continuation of: 

a) BEST administration to all AESL classes with TAG students 
in them; administer the BEST to both the TAG and the non-TAG 
students within two weeks of the classes' start and two weeks 
before they end; change forms from pretest to posttest and 
convert scores as directed in the administration manual. 
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b) AESL class observations at about the one-month, three- 
month, and five-month points ; particularly important are 
the observations of oral English usage levels, using the 
protocols DAC used and trained TAG staff in; QSCC might 
involve Umass grad students to participate in the 
observations an excellent idea, as long as a structure 
for quality maintenance is instituted. 

c) interviews with TAG students after they have received 
about two months of tutoring; TAG staff may wish to scale 
down the interview protocol if staff time is a problem; 
interview a randomly selected half of each group of 
tutees . 



For internal TAG Program management and development purposes, 
we recommend that the following be continued: 

a) BEST testing", as above; monitor the results after each 
testing period to see what the patterns of results say 
about the program and the students. 

b) AESL class observations, as for the PEP purposes 
above; review the results to see what insights for the 
prog-ram or the students are suggested. 

c) interviews with TAG students, as for the PEP purposes 
above; also, as in #2 a) & b) immediately above, 
synthesize and review for insights into program progress 
or problems. 



In addition, the observations of tutor training and interviews 
with tutors after they have been tutoring for two or more months 
would be valuable for the same types of ongoing internal evaluation 
purposes. These seem less critical, however, than the internal 
management recommendations detailed above. If there aren't 
resources available to conduct the tutor data collection as 
formally as DAC did in this past year's project, then perhaps 
periodic focus group meetings with tutors on some of the key themes 
from the interview protocol could serve the same purposes. 

Finally, the TAG application of the DEM may be useful for the 
staff's quarterly or semi-annual review and program analysis. This 
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would be useful for building and maintaining staff focus on the big 
program picture and how all the pieces fit together in it, for 
identifying areas that may be slipping out of focus before they get 
too far away, and for stimulating thinking about developments and 
improvements . 
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APPENDIX A 



DEMOGRAPHIC PROFILE OF TAG PARTICIPANTS AND COMPARISON GROUPS 

(March 1993 Data) 
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SELECTED DEMOGRAPHIC CHARACTERISTICS OF TAG AND COMPARISON GROUPS 

(Shown in Percents of Each Group) 





GROUP 




TAG 


LEVEL 1 


LEVEL 2 


LEVEL 2 




PARTICIPANTS 


WAITING 


WAITING 


ENROLLEES 


CHARACTERISTIC 


(N=82) 


LIST 


LIST 


(N=55) 






(N=337) 


(N=241) 




YEARS ON 










WAITING LIST 1 


57 


22 


30 


16 


2 


Q 

j 


18 


24 


9 


3 


34 


55 


46 


75 


AGE 16-24 


5 


5 


9 


4 


25-44 


54 


68 


80 


75 


45-59 


35 


22 


10 


16 


60 + 


6 


4 


1 


6 


YEARS OF 










FORMAL 










EDUCATION 0 


9 


8 


6 


7 


1-3 


7 


8 


5 


5 


4-6 


37 


29 


9 


29 


7-9 


34 


37 


39 


29 


10 + 


13 


18 


41 


29 


PRIOR ENGLISH 










STUDY (MONTHS) 










0 


59 


63 


15 


60 


1-3 


18 


16 


19 


9 


4-6 


9 


9 


22 


13 


7-9 


4 


1 


3 


0 


10 + 


11 


11 


41 


8 


YEARS AFTER 










U.S. ARRIVAL 










0-3 


77 


58 


50 


40 


4-6 


15 


26 


31 


38 


7-9 


2 


11 


12 


16 


10 + 


6 


5 


7 


5 


MARITAL 










STATUS : 










MARRIED 


85 


r-i r-f 

11 


73 


/ i 


SINGLE 


15 


23 


26 


27 


NUMBER OF 










CHILDREN 0 


37 


47 


59 


44 


1-3 


54 


45 


37 


47 


4-6 


6 


9 


3 


9 


7-9 


4 


0 


0 


0 


GENDER MALE 


29 


31 


22 


25 


FEMALE 


71 


69 


78 


75 



TAG Evaluation Final Report - 79 



APPENDIX B 

AESL CLASS SPOKEN ENGLISH OBSERVATIONAL ANALYSIS 
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We began the observation process with one special intent: to 
assess the confidence of the TAG "graduates" once they entered AESL 
Level 2 classes, since it was hypothesized by the program staff 
that they would be more confident in English usage. As a result, 
we observed for the English level of the students, rated each time 
they spoke a substantial "bit" in English. 

These English levels were rated on a 4-point scale, adapted by 
the authors from the BEST pronunciation scale, as follows: 

0 = Speech is almost always unintelligible. 

1 = Speech is frequently not comprehensible. 

2 = Speech is generally understandable, but occasionally 

difficult or impossible to comprehend. 

3 = Speech is readily understandable. 

This rating scale was taken as the next best indicator of 
TAG'S impact in lieu of direct confidence in speaking English. We 
had considered counting the number of times a student volunteered 
to speak in class, but ruled that out since whether a student 
volunteers or not in class is strongly influenced by the teacher's 
instructional approach at a given time some teachers call on 
students directly for responses, others open up questions for 
general class responses, some students are simply more or less 
vocal than others in the latter situations independent of their 
confidence in speaking English. 

To use the scale, the observer provided a rating each time a 
student spoke. After the class the ratings were averaged for each 
student. The evaluation design called for the observations to take 
place within the first month of the 6-month class term, near the 
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mid-point, and within the last month. At least one of the 
observation sessions was conducted by two independent observers 
using the same rating scale. 

Results 

Our first concern was with the reliability of the scale- On 
this count it performed fairly well, after observer practice. For 
the three observation periods with paired observers, we obtained 
Pearson correlations of: 0.81 (p <.01, 13 df) 

0.52 (p <.05 / 14 df) 

0.46 <.10, 13 df ) 
Second, to test the validity of the observation scale, we 
compared the observed ratings with one group of students' BEST Oral 
scores. They correlated at -0.13, a very low level, indicating no 
substantiation that this observed rating is a valid indicator of 
the students' oral English abilities. The BEST Oral exam is 
individually administered, however, and contains a variety of 
content, whereas the class observation was a group setting, and for 
each particular observation period the content was generally one or 
two topics. Both of those differences could account for student 
performance differences . 

On the other hand, the mean student ratings rose at a fairly 
steady rate over time in the class, as one would expect. 
Furthermore, the pre-ratings for the second term observations are 
again down to the vicinity of the pre-ratings from the first term. 
Data to show these trends are included in Table Bl. The results 
suggest that the scale is a valid measure of classroom oral English 
language performance. g< 
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Table Bl 

Mean Spoken English Observation Ratings -- AESL Level 2 Classes 



OBSERVATION 
POINT 


TERM 


FIRST 


SECOND 


CLASS 1 


CLASS 2 


CLASS 1 


CLASS 2 


BEGINNING 


1.22 (17) 


1.46 (12) 


1.5 (12) 


1.31 (13) 


MIDPOINT 


1.58 (15) 


2.18 (14) 






END 




2.09 (12) 







As a result, we conclude that the observation rating scale of 
English speaking ability used in this study is a moderately 
reliable measure of classroom-based English performance. The 
rating scale is not necessarily a good indicator of general English 
oral ability, however, because of its poor correlation with the 
BEST. 

Another feature of the instrument that showed up in the 
analysis was that bilingual Chinese observers rated the students 
slightly higher in English ability than did the English-only 
observers with whom they were paired. English-only and bilingual 
pairs were used in two of the three paired observations (the* 
correlations of 0.81 and 0.46) . The class means for each are shown 
in Table B2 . Note that while these differences were true for the 
class means, they did not occur on the individual student level 
that is, some individual students were rated higher by the English- 
only observers than by the bilingual observers . It seems likely 
that the English-only observers are more stringent in their 
expectations of clear and comprehensible English than the bilingual 
observers were; the latter, having come through a Chinese-speaking 
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environment and still living in it, find less clearly spoken 
English more comprehensible. The implications are that the 
absolute value of eny observed ratings of spoken English be 
interpreted in light of the observer's English speaking background, 
and that any planned future observations take the observers' 
backgrounds into account . 



Table B2 

Mean Spoken English Observation Ratings 
by Observer Language Background 



OBSERVER 


CLASS 1 
(MIDPOINT) 


CLASS 2 
(MIDPOINT) 


N = 15 


N = 14 


ENGLISH ONLY 


1.47 


2.08 


BILINGUAL 
(ENGLISH/CHINESE) 


1.68 


2.27 



TAG "Graduates" Compared to Others 

The mean ratings for each observation period for the TAG 
"graduates" versus the others in the classes are shown in Table B3 . 
Note that the comparisons are tenuous since the numbers of TAG 
students are very small in some of the classes. The trend, 
however, is. that the students progress at about the same rate in 
both classes regardless of whether they were TAG students or not. 
Moreover, the differences between the TAG and the other students in 
a given class is rarely beyond a few tenths of a point on the 4- 
point scale, which is essentially the same given the reliability 
level of the rating scale. We take this as evidence that the TAG 
students do perform as well in the AESL Level 2 setting as those 
students who have not had TAG. (Inferential statistical tests are 
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inappropriate given the small numbers of comparison students in 
each of the classes. It would also have been inappropriate to 
combine both classes to compare all TAG students to all others in 
order to have greater cell sizes, since the observations were not 
necessarily taken at the same time across classes, the class/ 
teacher effect could be substantial, and different raters were 
used. ) 

Table B3 

Mean Spoken English Observation Ratings 

for TAG vs non-TAG Students 
(Number of Students shown in Parentheses) 







- 


OBSERVATION POINT 


TERM 


CLASS 


GROUP 


BEGINNING 


MIDPOINT 
(Observer 1) 


MIDPOINT 
(Observer 2) 




1 


TAG 


0.9 
(3) 


1.3 . 
(1) 


1.0 
(1) 






non- 
TAG 


1.3 
(14) 


1.5 
(14) 


1.7 
(14) 


FIRST 


2 


TAG 


* 


2.1 
(11) 


2.3 
(11) 






non- 
TAG 


* 


2.1 
(3) 


2.3 
(3) 




1 


TAG 


1.3 

(5) 










non- 
TAG 


1.3 
(5) 






SECOND 


2 


TAG 


1.6 

(9) 










non- 
TAG 


1.3 
(3) 







* = The data tor this class were not usable. 
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This analysis of observational data suggests that it served as 
a valuable pilot for developing a sound observational scheme of 
oral English performance in the classroom. That is, there appears 
to be very little, if any, observational data of this type in the 
adult ESL assessment literature. We have gathered data to suggest 
that the scheme shown is reliable and valid. With greater degrees 
of control . in implementing the design in other larger-scale studies 
(i.e., piloting and practice before the actual data collection, 
careful pairing of observers, similar numbers of treatment and 
comparison group members, etc.), this simple observational scheme 
may be useful for evaluation of -such programs. 
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APPENDIX C 
TAG PARTICIPANT RETENTION DATA 
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We examined the TAG participant files to extract information 
which would help determine the extent to which the two main 
objectives were met for Goal #2 of the Discrepancy Evaluation 
Model's application to TAG. The results are as follows. Records 
were available for 109 TAG participants placed as of July 23, 1993. 



Objective A: To retain 85 percent of the TAG learners in the 
program for at least one year. 

We first examined those TAG learners who were placed in 

tutoring before 7/24792, so that they would have the potential of 

retention in the program for at least one year. There were 57 

placed for TAG tutoring before 7/24/92. Of those 57 students, 88 

percent (50 students) were either: 

a) still in TAG (19 students), 

b) had entered an AESL Level 2 class in July 1992 or January 
or July 1993 (31 students), or 

c) had unusual circumstances which made them meet the intent 
of this objective (7 students) . 

The 31 students who entered AESL Level 2 classes were 
generally in TAG tutoring for less than one year. But since they 
met the intent of this objective, they are counted toward it. In 
fact, the average time between their TAG tutoring start and AESL 
class entrance was 8.9 months. This average is undoubtedly larger 
than the time it really took them to "complete" TAG, for the 
classes begin only in January and July and a TAG student who 
completes TAG and qualifies for Level 2 in, say, May, must wait 
until July to enter a class. Thus, the additional months of 
waiting after qualifying for Level 2 have been added into the 
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average above. (Dates at which qualification for Level 2 was 
reached are not available in the records.) 

The circumstances for the group who technically were not 
retained in the program for at least a year, but who are considered 
to have met this objective, include the following: 

- stopped in the middle of the year and resumed TAG in 6/93 

- finished all the tapes, but couldn't schedule a class until 
7/93 

- qualified for Level 2, can't schedule a class because of work 
schedule 

- qualified for Level 2; going to China for one month; will 
enter a class^in 1/94 

- was 13 months before being able to schedule a class 

- tutor went to China for one year; students insist on waiting 
for her to return to resume TAG (2 students) . 

The above figure of 88 percent exceeds the standard stated in 

the objective. Thus, we can conclude that this objective has been 

achieved and exceeded. 

Objective B: 50 percent of the TAG learners retained in the program 
will reach Level 2 within one year. 

We counted those students who were either (a) in TAG for one 

year or more without termination, or (b) qualified for Level 2 and 

either enrolled in an AESL class, were noted as waiting for a class 

which met their schedule, or did not want to enroll in a class (and 

generally were waiting for higher level tapes to become available) . 

There were 58 students who met the criteria above. Of those, 48, 

or 83 percent reached Level 2 within one year. Thus, this 

objective is achieved well beyond its expected level. 
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